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ABSTRACT 

Georgia's Reading Challenge program is intended to provide a 
quality after-school program for students in grades four through eight with 
opportunities to improve student reading skills and to enhance student 
interest in reading. The goals of Georgia's Reading Challenge program for 
1998-1999 were to increase levels of reading achievement; improve school 
attendance; improve academic performance; increase participation by students 
in supervised academic, educational, community service, or other special- 
focused activities; and provide students meaningful, enjoyable after-school 
activities. The purpose of this evaluation is to describe the first year of 
implementation for Reading Challenge and provide preliminary information that 
can be used for decision making regarding the program. Three questions guided 
the evaluation: how Reading Challenge is being implemented; what impact 
Reading Challenge has on student reading achievement and academic 
performance; and to what extent the type of program impacts the level of 
implementation and student participation. In terms of implementation, the 
data show that, for the most part, the program is being implemented as 
planned. In terms of academic achievement, the preliminary results are mixed. 
It is too early to determine the overall effects of the program. Two 
appendixes contain: Methodology; Protocol for Pre-Observation Teacher 
Interview; Guide for Classroom Observations ; Observation Checklist; Protocol 
for Group Discussion/Interview; Readiness Challenge Survey; and Reading 
Challenge Program Guidelines. (Contains 16 charts and 1 table.) (PM) 
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Georgia’s Reading Challenge program is intended to provide a quality after-school 
program for students in grades four through eight with opportunities to improve student 
reading skills and to enhance student interest in reading. The goals of Georgia’s Reading 
Challenge program for 1998-1999 were to increase levels of reading achievement; 
improve school attendance; improve academic performance; increase participation by 
students in supervised academic, educational, community service, or other special- 
focused activities; and provide participating students meaningful, enjoyable after-school 
activities. During the 1998-1999 school year, its first year of statewide operation, 
Reading Challenge was implemented at 163 schools and private organizations throughout 
Georgia. 

The purpose of the evaluation is to describe the first year of implementation for Reading 
Challenge and provide preliminary information that can be used for decision making 
regarding the program. Three questions guided the evaluation: 

1 . How is Reading Challenge being implemented? 

2. What impact does Reading Challenge have on student reading achievement 
and academic performance? 

3. To what extent does the type of program (i.e., school based or private) impact 
the level of implementation and student participation? 

The data collected as part of this evaluation came from three different sources of 
information: structured observations of a sample of Reading Challenge programs, focus 
groups conducted with key program personnel, and surveys administered to each Reading 
Challenge program during the spring of 1999. Fifth and eighth grade Reading Challenge 
students’ Iowa Tests of Basic Skills (ITBS) scores were also examined as part of the 
evaluation. Data from the observations, focus groups, and surveys were triangulated to 
add confidence and credibility to the study. 

Data from the year one evaluation are descriptive in nature and focus primarily on the 
first evaluation question. Preliminary information on the second and third evaluation 
questions are, however, provided in this report. These questions will be more fully 
answered in year two of the evaluation. 

In terms of implementation, the data show that, for the most part, the program is being 
implemented as planned. There were slight differences in implementation between 
public and private sites. For example, there was variation in the criteria used to select 
students. Public sites tended to select students with the greatest academic need, while 
private sites tended to use a lottery system or target a particular group. Slight differences 
in attendance and reading achievement were also found between public and private sites, 
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however, these differences were either not significant or can only be understood through 
a more long term examination. The data suggest that both public and private Reading 
Challenge sites had difficulty in retaining students. 

In terms of academic achievement, the preliminary results are mixed. Overall, students 
attending private programs had slightly higher ITBS scores in fifth grade, however, 
students attending public programs showed higher ITBS scores in the eighth grade. When 
scores were assessed relative to the length of time in the program, the results were mixed 
between fifth and eighth grades and between public and private facilities. Without pre- 
implementation data, which show how students performed before entering the program, it 
is difficult to understand what these results might mean. Finally, teachers' perceptions of 
students' academic growth were positive. Teachers felt that students' academic 
performance is improving as a result of the program. The data collected also resulted in a 
number of specific recommendations for program improvement. These recommendations 
are presented in Section IV of this report. 

It is too early to determine what the data means in terms of the overall effects of the 
program. The statewide evaluation of Reading Challenge will continue during the 1999- 
2000 school year. Data will be collected on both new Reading Challenge programs and 
those receiving their second year of funding. Both fall and spring surveys will be 
administered to various stakeholders within the program (i.e., teachers, students, parents) 
and more demographic information will be collected. This will allow Georgia 
Department of Education (GDOE) to develop a broader perspective of the program and 
more fully examine the program’s effect on achievement. 
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Part I: 

Reading Challenge Program Overview 

Georgia’s Reading Challenge program is intended to provide a quality after-school 
program for students in grades four through eight with opportunities to improve student 
reading skills and to enhance student interest in reading. Reading Challenge programs 
are designed to be motivating and enriching for participating students. The program 
provides opportunities for student recreation, socialization, academic enrichment, 
community service, and/or specialized study which gives students the opportunity to put 
their reading and writing skills into practical, meaningful use. Both the students’ parents 
and their classroom teachers should be an integral part of the Reading Challenge 
program. 

During the 1998-1999 school year, its first year of statewide operation, Reading 
Challenge was implemented at 1 63 schools and private organizations throughout Georgia 
(43 private sites, 120 public sites). The first year goals of the Reading Challenge 
program were: increasing levels of reading achievement; improving school attendance; 
improving academic performance; increasing participation by students in supervised 
academic, educational, community service, or other special-focused activities; and 
providing participating students meaningful, enjoyable after-school activities. 

Purpose of the Study 

As part of the State Superintendent of School's initiative to evaluate all statewide 
educational programs, the GDOE is conducting a three-year statewide evaluation of the 
Reading Challenge program. This evaluation is not intended to be an evaluation of 
individual programs, but an evaluation of the program statewide. The evaluation 
questions are as follows: 

1 . How is Reading Challenge being implemented? 

2. What impact does Reading Challenge have on student reading achievement 
and academic performance? 

3. To what extent does the type of program (i.e., school based or private) impact 
the level of implementation and student participation? 

This first year report is descriptive in nature and focuses primarily on the first evaluation 
question. From an evaluative standpoint, more substantive information on program 
effects on achievement and the impact of different program types is difficult to ascertain 
with one year or less of program implementation. Preliminary information on the second 
and third evaluation questions are, however, provided in this report. These questions will 
be more fully answered in year two of the evaluation. 
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Methodology 

The data collected as part of this evaluation came from three different sources: structured 
observations of a sample of Reading Challenge programs, focus groups conducted with 
key program personnel, and surveys administered to teachers and administrators in each 
Reading Challenge program in the spring of 1999. Additionally, ITBS scores for fifth 
and eighth grade students were examined as part of the evaluation. 

The GDOE contracted with the Occupational Research Group (ORG) in the School of 
Leadership and Lifelong Learning, College of Education, at The University of Georgia to 
conduct structured observations and focus groups at 15 Reading Challenge sites. The 
sample of Reading Challenge sites were selected with input from the staff in the 
Curriculum and Reading Division of the GDOE. This approach was used to obtain a list 
of sites that represented the diversity within the program and enabled the evaluation team 
to obtain detailed information on each program. The shortcoming of this approach is that 
the use of a non-random sample reduces the generalizability of the findings. The 15 
Reading Challenge observation sites were stratified across the five geographic regions of 
the state (i.e., north, south, east, west, and metro). A total of 29 classroom observations 
(14 at private sites, 15 at public sites) occurred at the 15 Reading Challenge sites. Focus 
groups were conducted at 13 of the 15 Reading Challenge observation sites. 

The surveys were designed and administered by the Research, Evaluation, and Testing 
Division of the GDOE. Each Reading Challenge program was asked to fill out a survey 
in the spring of 1999 that described the activities occurring at their sites. 

A triangulation of methods of data collection was used to provide the GDOE with data 
from multiple sources. This technique is intended to give more confidence and 
credibility to the study. A detailed description of the methodology used in this study, the 
observation and focus group protocols, and the survey instruments are provided in 
Appendix A. 
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The cornerstone of the Reading Challenge program is the provision of meaningful after 
school activities with a specific focus on reading. This section of the report addresses the 
first evaluation question by discussing how the program was implemented at the Reading 
Challenge sites. This section is divided into two parts. First, the program's guidelines are 
discussed. Second, the extent to which Reading Challenge sites implemented the 
program as planned is examined. The latter portion also highlights differences in 
implementation across the sites, and where applicable, makes specific comparisons 
between the public and private programs. 

The information provided in this section of the report is based on data collected during 
site visits to 15 Reading Challenge programs, focus group discussions conducted at 13 of 
the 1 5 observation sites, and surveys administered to teachers and Reading Challenge 
administrators at each of the 163 Reading Challenge programs. Throughout the 
presentation of the findings, private Reading Challenge observation sites are identified as 
A through H and public Reading Challenge sites are labeled AA through GG. 

Program Guidelines 

According to Georgia ’s Reading Challenge Program Guidelines (see Appendix B), the 
purpose of Georgia’s Reading Challenge program is to provide a quality after-school 
reading program for students in grades four through eight with opportunities to improve 
student reading skills and to enhance student interest in reading. The goals of Georgia’s 
Reading Challenge for the 1998-1999 school year were to: 

• Increase levels of reading achievement for participating students. 

• Improve school attendance of participating students. 

• Improve academic performance of participating students. 

• Increase participation by students in supervised academic, educational, 
community service, or other special-focused activities. 

• Provide participating students meaningful, enjoyable after-school activities. 

The program is funded by the State of Georgia through the GDOE, and is offered to 
agencies with either public or private affiliations through a competitive grant process. 
Reading Challenge funding may supplement existing after-school programs or may 
establish new programs. The location of the program is determined by the needs of the 
community and the types of facilities available. A school building, local or civic 
building, or private facility may be used as long as liability coverage is provided and the 
building meets all local, state, and federal codes and guidelines for an after-school, child- 
care facility. Each Reading Challenge after-school program must have an advisory board 
responsible for setting policies and guaranteeing that program guidelines are followed. 
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Reading Challenge programs are required under state guidelines to operate three hours 
per day, five days per week for a minimum of 130 days per year (26 weeks or 
approximately 6.5 months). The adult/student ratio is a maximum of 1 to 20 for overall 
program activities and supervision. Certified teachers provide students a minimum of 
four hours per week of small group reading instruction, during which the teacher/student 
ratio is not to be more than one to ten. Reading Challenge programs must serve a 
minimum of 20 students and a maximum of 60 students. All students in grades four 
through eight are eligible to participate in Reading Challenge, and students are served 
regardless of race, color, creed, gender, national origin, or handicapping conditions. 
Georgia's Reading Challenge programs are to be offered free of charge to participating 
students. 

Instruction for Reading Challenge must be geared to the educational needs of each 
student as determined by the Basic Literacy Test (BLT) 1 or the child's classroom teacher 
at school. All instructional materials are research- based and proven effective with 
students. Computers are available for student use at Reading Challenge sites and 
technology is used to remediate, reinforce, or enrich individuals' reading skills. In 
addition, students borrow books from the school or public library and keep a record of all 
books read. The original guidelines also required two forms of reading assessment: (1) 
norm-referenced (ITBS); and (2) periodic informal pretest and posttest assessments. The 
ITBS requirement was later discontinued due to the fact that students enter and exit the 
program at different points throughout the year, and to avoid conflicts with the state 
testing program. (Fifth and eighth grade students might well have been required to take 
portions of the ITBS as part of Reading Challenge as late as December or January, and 
then again for the state assessment three or four months later.) 

In addition to the focus on reading, Reading Challenge programs provide opportunities 
for student recreation, socialization, academic enrichment, community service, and/or 
specialized study which gives students the opportunity to put their reading and writing 
skills into practical, meaningful use. Both the students’ parents and their classroom 
teachers from their home schools are to be an integral part of the Reading Challenge 
program. Parents must be informed of student progress and/or program activities on a 
monthly basis, either in writing or through a conference. Classroom teachers of 
participating students are to be involved in planning for student programs and are to be 
informed of student progress on a monthly basis. 

Other Reading Challenge requirements include specific staff qualifications, staff 
development, funding and reporting specifications, a written discipline policy, and a 
policy for student attendance. 



1 The BLT provides an oral assessment of alphabet recognition, phonics knowledge, sight word 
recognition, and reading comprehension. Points are assigned based on skills achieved in each section of 
the test, and these scores can be interpreted to show each student’s strengths and weaknesses in reading. 
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Program Implementation 

In assessing the extent to which sites have adhered to the program guidelines in 
implementing the Reading Challenge program, several areas were observed and 
addressed in the focus groups and surveys. This section focuses on the extent to which 
Reading Challenge program staff understand and have put into practice the Reading 
Challenge philosophy and structure. 

Purpose and Goals of the Program 

Focus group participants were asked what they believed to be the purpose and goals of 
the Reading Challenge program. Most participants indicated the purpose was, in the 
words of one teacher, "...to increase reading achievement of all students and to foster a 
love of reading in participating students. ” 

When asked about the goals of the program, there was a great deal of variation ranging 
from the notion of providing students with meaningful, after school activities to 
improving school attendance, providing homework assistance, reducing discipline 
referrals, and improving communication. Teachers' perceptions of the purpose and goals 
of Reading Challenge are, for the most part, more narrow than those stated in the 
program guidelines. Generally, teachers believe the program should increase students' 
reading interest and ability, as well as improve their self-esteem. Most programs report a 
robust list of extracurricular activities, but only some participants note these activities as 
a specific goal of the program. Even fewer participants linked the goals of Reading 
Challenge to general academic performance and school attendance. 

Many focus group participants reported that it had been difficult to implement all of the 
program’s goals as stipulated in the guidelines given all of the requirements and 
expectations and the short time frame between funding and implementation. Staff at both 
the public and private sites shared these concerns. Private Reading Challenge programs, 
particularly newly formed programs, indicated a greater need for overall guidance. One 
participant expressed this sentiment in this way: “ There needs to be more explicit and on- 
going guidance and training provided for programs of this sort.... There is a need for 
ideas of how to successfully design and implement a successful Reading Challenge 
program. ” 

Some participants recommended that the program could use “objective based goals,” in 
which individual programs would have more discretion to tailor their programs to the 
local needs. A typical observation was, “ We feel that more discretion needs to be given 
to the program director while guidance is given by setting objectives; teachers at each 
site may have different and individual needs, and that these needs should be taken into 
consideration in program implementation. ” 
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The field observers at both public and private sites suggested that program guidelines 
need to place more emphasis on the integration of children’s literature into reading 
instruction and developing curriculum guidelines for reading instruction in the Reading 
Challenge program. 

Study participants at several sites recommended that the Reading Challenge program 
guidelines include appropriate planning or lead time prior to actual program 
implementation. Participants in the group discussions noted the need for more planning 
time between grant award and the start-up of the Reading Challenge program. 

Generally then, teachers perceive the purpose and goals of the Reading Challenge 
program more narrowly than indicated in the state guidelines. This seems to reflect the 
diversity in the sample of programs observed across the state and the extensive teacher 
and administrator surveys. Interestingly, few teachers mentioned extracurricular 
opportunities or home school academic performance and attendance as goals of the 
program for them. 

Program Type 

Reading Challenge programs are funded for operation in organizations with either public 
or private affiliations. Programs could be added to existing after-school programs or be 
newly created. Of the 15 Reading Challenge programs observed, ten were developed as 
new and separate programs. Seven of these new programs were developed at public 
schools and three were at private facilities. The other five Reading Challenge programs 
that were observed were developed as extension programs. All of the “add-on” programs 
were organized through private organizations. 

Program staff at the extension programs generally agreed that the addition of reading 
instruction and the focus on academics significantly enhanced the previous programs. 

For example, one teacher said, "The existing structure of the after-school program prior 
to the implementation of Reading Challenge lacked opportunities for supervised 
academic assistance. With the implementation of Reading Challenge , a specific focus for 
meaningful reading instruction was added. " 

Although approximately one-half of the observation sites were Reading Challenge 
programs with private affiliations, it is important to mention that overall, private 
programs comprise only 26 percent of the total number of Reading Challenge sites. 

Chart 1 shows the percentage of public and private Reading Challenge centers. 



2 A non-random selection process was used to assure that the observation sites reflected the diversity of 
practices found throughout the Reading Challenge program. 
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Chart 1: Percentage of Programs Identified as Public or Private 



74% 




□ public 

□ private 



Student Characteristics 



Reading Challenge programs are designed to serve any student in grades four through 
eight. Each participating site, however, has considerable autonomy in deciding the 
specific configuration of their Reading Challenge program. Of the sites observed, both 
public and private, there was considerable variation in the grade levels being served. For 
instance, four of the fifteen observation sites served grades four through eight; two served 
grades six through eight; three served grades four through six; four programs served only 
grades four and five, and two programs served grades two through five. 

There is also a great deal of diversity at both the public and private Reading Challenge 
programs in criteria used to select students for enrollment. For example, one private site 
targeted a predominately Hispanic, transient population while another served students 
who are members of the church in which the program is housed. Many of the school- 
based programs reported attracting both low and high performing students into the 
program. For example, one teacher noted, "We are working with all different levels and 
reducing discipline referrals and increasing interest in reading; the program helps kids 
catch up and provides enrichment for the gifted. The object of the program is to improve 
the reading skills of all students. " 

Overall, each program provider develops criteria they feel is appropriate in selecting 
students for participation. Programs have autonomy in choosing enrollment criteria. 
Focus group participants believed that this type of autonomy allowed them to target 
students not ordinarily eligible for special programs. For instance, one participant said, 
“Most of the time when we apply for a grant, it is just for the at-risk child. It is 
wonderful that this program is not just for the at-risk child. It allows us to extend it to all 
children. Neither the at-risk nor the top student has suffered in this program. It helps the 
higher student have empathy and be more understanding of other kids. We recognize 
everybody for doing something well " 

In many programs, students are selected via a lottery system. Other programs, 
particularly those that are school-based, reported using student’s ITBS scores as their 
selection criteria to allow them to concentrate on students with academic needs. While 
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some programs serve both gifted and learning disabled students, other programs target 
students that are performing below grade level. For instance, “Reading programs often 
meet the needs of the on-grade level child. It would be great if the program really 
emphasized the lowest level child the most. Programs should have one hundred percent 
participation from those students who are below the 50th percentile on the ITBS. ” 

Several focus group participants felt that Reading Challenge needs to “target children 
who struggle, not randomly selected applicants as now,” and “should only be for students 
who have reading problems.” They noted that the “lottery system used [in some schools] 
meant that some children who really need the help did not get selected to participate.” 
Student recruitment strategies used by Reading Challenge programs varied across sites. 

At public sites, programs typically mailed flyers announcing the program to parents or 
sent them home with students. Some parent-teacher organizations advertised the new 
program. Recruitment for programs at privately managed sites was handled a little 
differently. For example, recruitment at one program was handled jointly by school 
personnel and the YMCA, while another recruited using newspaper and radio 
advertisements, as well as flyers in the elementary and middle schools. 

The survey data show that, overall, public sites have more students enrolled per grade 
than private sites. Enrollment distribution across grades indicates that Reading Challenge 
students cluster toward the lower grades. Chart 2 shows the average number of students 
enrolled in public and private Reading Challenge programs by grade. 

Chart 2: Average Number of Students Enrolled by Grade 



20 

15 
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The site surveys also suggest that males and females participate in the program at about 
the same rate. However, African American students comprise almost two-thirds of the 
children served by the program. Table 1 shows the demographic characteristics of 
students enrolled in Reading Challenge. 



Table 1 : Student Participants by Gender and Race 





African American 


White 


Other Ethnicity 


Male 


29.5% 


16.9% 


1% 


Female 


35.7% 


15.6% 


1% 



*does not equal 1 00 due to rounding 
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Overall, there appears to be a great deal of diversity in the characteristics of Reading 
Challenge students. The observation and focus group data show that the configuration of 
grade levels served by the programs varies by site as well as the criteria used to select 
students for program participation. Ethnically, there is a high proportion of African 
American students across all programs. Finally, public sites, which comprise 74 percent 
of all Reading Challenge programs, also serve more students per site than programs with 
private affiliations. 

Staffing 

Reading Challenge guidelines stipulate that programs must employ certified teachers to 
conduct individual and small group reading instruction for four hours per week. Nearly 
all of the Reading Challenge programs observed were staffed with teachers who are also 
full-time employees of area school districts. Focus group participants saw this as both an 
advantage and a disadvantage. Teachers are familiar with the school facilities and the 
resources available, and they often have access to school materials such as computers and 
software. Also, teachers know some of the students who are enrolled in the Reading 
Challenge program and this seems to foster communication with students' classroom 
teachers. Further, there is little or no extra travel for these teachers. However, many 
focus group participants reported that fatigue and burnout are problems at sites where 
teachers work all day in addition to Reading Challenge. One observation site employed 
three certified teachers who are not working elsewhere. 

Some programs have been creative in their approach to staffing to account for teacher 
workloads. For example, at Site AA, several teachers are employed and work on a 
rotation basis (two teachers per day) and teach classes of special interest to them, such as 
science, language arts, video production, drama, using the Internet, creative writing, and 
literature groups. 

One or more paraprofessionals were employed at nearly every Reading Challenge site 
that was observed. Paraprofessionals’ duties included monitoring computer use by 
students, supervising socialization time and recreation activities, and conducting lessons 
under the supervision of a lead teacher. 

Volunteers also help at several Reading Challenge programs. Volunteers were high 
school or college students, parents, senior citizens, or other community members. At 
Site A, a church facility, senior citizen church members serve as mentors to the Reading 
Challenge students. High school students volunteer as part of their community service 
commitments at Site BB, where they monitor and read to students. Local college 
students, made available through a scholarship program, help with recreational and 
extracurricular activities such as dance, piano, arts and crafts at Site E, a Boys' Club. 
Volunteers at Site H recruited from the church that houses the program. Parents help out 
occasionally at several sites. For example, one mother at Site C provides students with 
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baked goods. Parents and volunteers are quite visible at Site BB, where parents read to 
the children and help with projects. However, parents are generally unavailable because 
of time constraints and other responsibilities. 

Some sites report using school personnel and community leaders as volunteers. At Site 
CC, school administrators and other community leaders read to the children. At Site GG, 
classroom teachers participate in the program as volunteers on an irregular basis. Site CC 
reports "the superintendent, principals, and other community leaders have been requested 
to read to the children." 

There was consensus across public and private sites with regard to the responsibilities of 
Reading Challenge teachers. For example, one site observation report states that 
“Teacher responsibilities are to conduct assessments of students periodically during the 
program sessions, develop and implement lesson plans related to reading instruction, 
assist students in developing and maintaining reading response/creative writing journals, 
and provide individual reading instruction based on individual student reading needs.” 

Some programs (both public and private) reported having organized lesson planning for 
Reading Challenge. At Site D, for example, weekly lesson plans developed by the lead 
teacher are kept in a weekly plan book and available to all personnel. According to one 
teacher at Site GG, Reading Challenge "teachers meet regularly to plan and communicate 
the progress of each student, and to determine if the learners need to be regrouped." In 
fact, observers at that site report that "based on information received from classroom 
teachers, each Reading Challenge teacher designs specific lessons to address the needs of 
the individual child." Lesson planning, however, seemed to be problematic for some 
Reading Challenge teachers. Some sites indicated that there was little time for lesson 
planning and that attendance problems made it particularly difficult for teachers to carry 
out planned activities. 

Instructional Practices 



During the observations and focus groups, evaluation staff examined the instructional 
practices teachers employ as part of the Reading Challenge program. Teachers and 
administrators agreed that their instructional practices include a mix of direct instruction 
(individualized and group), projects, and extracurricular activities. For example, 
"...teachers use cooperative learning groups, use partner reading, utilize seat work in 
groups of two or three to work on specific skills, use problem-solving activities and 
higher order thinking skills; they focus on using higher questioning skills such as 
drawing inferences, using lengthy explanations, and allowing students to ask questions of 
teachers and each other. " 

Although reading is the focus of each program, sites approach reading instruction in 
many ways. At SiteAA, a teacher says "foci include video production, drama, using the 
Internet, and writing," adding that "in each case, reading is incorporated into these small 
groups." 
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Several Reading Challenge teachers use creative instructional practices to enhance 
reading skills and encourage reading. Games are used to reinforce concepts in 
comprehension, phonics, sight words, vocabulary, listening, and remembering. Students 
at Site A practice sight words and sentence construction using a Bingo game. One teacher 
plans occasional scavenger hunts on the Internet for her students. Another teacher, at Site 
CC, incorporates games such as Boggle, Scrabble, and Monopoly into reading activities. 
Other creative approaches include team assignments, writing plays, and story writing 
activities. At Site F, a teacher uses "buddy assignments" in which students work in pairs, 
grouped according to level of task difficulty or individual student interest. Students at 
Site BB wrote a play extending a scene in a book they were reading. These students also 
made costumes, props, and memorized lines for the play. Another activity at Site BB was 
a creative writing activity called "Stamp-a- Story," in which students randomly stamped 
pictures across the top of a page, then wrote a story to fit the pictures. 

Focus group participants indicated that there are a number of areas where more guidance 
is needed to better meet the needs of students. They are: 

• Provide instruction in higher level comprehension. 

• Emphasize writing process instruction, including pen pals. 

• Incorporate multi-media art projects and music. 

• Offer cooperative learning activities. 

• Integrate more children’s literature into instruction. 

• Develop thematic study approaches to instruction. 

• Provide instruction for composing poetry, plays, and stories. 

• Teach comprehension strategies including story maps, semantic webbing, and 
prediction. 

• Develop word banks and word walls for reading. 

• Integrate science and math. 

• Incorporate more cultural diversity. 

• Implement community activities that require reading. 

Many focus group participants expressed the belief that Reading Challenge teachers have 
to make an extra effort to ensure that the program will hold students’ attention. One 
participant put it this way: “Because Reading Challenge is after school, it makes for a 
long day. Extra care needs to be given in planning lessons that are stimulating. More 
thoughtful planning including additional enrichment activities incorporating music, art, 
and writing is needed.” 

Given that the cornerstone of the Reading Challenge program is the emphasis on reading, 
the amount of time programs spent on reading instruction was examined as part of the 
evaluation. The data show that in both public and private programs, students reportedly 
spent at least 45 minutes per day on small group reading instruction. However, private 
sites reported spending slightly more time on small group instruction than public Reading 
Challenge programs. The data are presented in Chart 3. 
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Chart 3: Average Number of Minutes Per Day Spent on Small Group Reading 
Instruction by Grade 
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Observers in classrooms at Site C note that “ direct instruction was being used as part of 
the reading recitation exercise in three of the classrooms, ” adding “ individualized direct 
instruction was offered in another [classroom].” Observers at Site E note that, in one 
classroom observation, the teacher gave “ explicit directions on how to brainstorm and 
gather responses.” 

Reading Challenge programs were also asked to indicate the amount of time they spent 
each day on other types of reading instruction (not small group). The data show that both 
public and private sites spent at least 20 minutes per day on other types of reading 
instruction. Consistent with the data discussed above, private sites reported spending 
more time on “other” reading than public programs. The data are presented in Chart 4. 

Chart 4: Average Number of Minutes Per Day Spent on Other Reading Instruction 




In terms of instructional practices, Reading Challenge teachers employ creative strategies 
to teach reading and maintain student’s interest. However, focus group data clearly 
support the need for more guidance in assisting teachers in developing a more 
comprehensive curriculum. Finally, both public and private sites reported spending more 
than the required amount of time (1 hour each day) on reading and related activities. 
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Extracurricular Activities 



A major component of the Reading Challenge program is the provision of enriching 
extracurricular activities. Overall, the Reading Challenge programs in this study reported 
numerous activities beyond reading. These activities included: access to library 
facilities, help with homework, socialization, recreation, community service, and special 
events. 

Several observation sites reported weekly trips to the public library as an integral part of 
the program. For instance, students at Site G go to the public library every week, and 
Site BB makes two visits each week to the school media center. 

Some Reading Challenge programs also indicated that a specific block of time for 
students is available for homework assignments. These blocks of time varied from 1 5 
minutes to one hour. Teachers or paraprofessionals assist students with homework as 
needed. For example, at Site AA, a portion of small group reading instruction was 
redirected to homework help after requests from parents. 

Socialization and recreational activities are also a component of the Reading Challenge 
program. Private sites, specifically those that added Reading Challenge to existing 
programs, seem to offer the widest variety of recreational activities. Activities varied 
from art, chess, and computers to sports activities such as basketball and archery. 

Reading Challenge participants are also encouraged to perform community service 
activities as part of the program. Some of the community service activities included 
sorting books at the public library, visits to nursing homes, and a beautification project at 
a school where the program is held. While some programs did indicate having a 
community service component in place, other programs felt that this component of the 
program needed to be improved. 

When asked to indicate the number of days per month students spent doing community 
service activities, overall, private sites reported spending more time doing this activity. It 
may be that programs with private affiliations, particularly pre-existing afterschool 
programs, have already developed relationships with entities that support community 
service. Also, across sites, the amount of time spent on community service increased 
with the grade of the students. One plausible explanation for this may be that older 
students are beginning to assume more responsibility in general and benefit more from 
giving back to the community. 
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Chart 5 shows the average number of days Reading Challenge students spent doing 
community service by grade. 

Chart 5: Average Number of Days per Month Spent doing Community Service 
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Reading Challenge programs provide special events and programs for students. At 
Site E, field trips related to units of study are taken on Fridays. Site FF also provides 
students with field trip opportunities. Students at Site CC made their own kites. These 
students also held a special Mothers' Day Social for parents and grandparents. Site C has 
hosted special events such as a black history program, a Cinco de Mayo (Mexican 
Independence Day) program, and an etiquette program. 

Participants in the study noted various enrichment opportunities, including experience 
with computer technology, field trips, and special events. Participants at Site BB say the 
program has "given kids who might not have been involved in extracurricular activities 
an outlet." Some enrichment opportunities were unexpected, but considered to be 
strengths of the program. For example, given the ethnic diversity among Reading 
Challenge students at Site A, “students have become more sensitive to other cultures” as 
pointed out by participants in the group discussion. 

For programs with active community service components, the impact of Reading 
Challenge on the school and community is considered a strength of the program. Focus 
group participants agreed that, “ Reading Challenge opens the schools to the community; 
it allows for greater use of the school and its facilities. " 

Programs that feel the need for more community involvement suggested coordinating 
activities with local recreation department programs. A representative comment was, 

“ Community activities that require reading and/or research skills in content areas would 
improve reading and academic skills and enable students to make connections to real 
life. ” 

In summary, Reading Challenge programs provide a variety of enrichment opportunities 
for students. Activities that are offered include assistance with homework, recreation, 
socialization, field trips, and community service. Private sites seem to have more 
community service activities available to students. It is not surprising then that these 
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students reportedly spend more time doing community service than public Reading 
Challenge students. 

Technology 

The majority of the Reading Challenge programs observed (87 percent) use technology, 
particularly computer programs, to supplement individual instruction. Technology is 
considered a strength of the Reading Challenge program. According to participants at 
Site B, individualized computer programs allow students to work at their own pace. 
Participants at SiteAA claim the program “has provided a venue to teach students about 
the Internet using the resources already existing in the school .” At Site EE, students use 
the Title I computer lab in addition to another technology center in the school for learning 
centers. 

Public and private sites reported spending an average of 25 to 60 minutes per day on 
computer-based learning. Private Reading Challenge programs spent a significant 
amount of time on this activity with sixth and seventh grade students as compared to their 
public counterparts. The reason for this discrepancy is unclear. Chart 6 shows the 
average amount of time per day spent on computer-based learning. 

Chart 6: Average Minutes per Day Spent on Computer-based Learning by Grade 




Some sites reported little use of computers, however at Site E, computers are used mainly 
for the Accelerated Reader program and for teacher management of the Reading 
Challenge program. There is little use of computers and software for Reading Challenge 
beyond word processing. Site DD does not use computer technology as a component of 
the structured curriculum. Rather, computer use is offered as a choice for students who 
finish work early or during certain activity periods. Programs include comprehension/ 
language arts-related games. Field observers at Site G report that no reading management 
software, ancillary materials, or vocabulary programs from the state list are used, and the 
software used for enrichment does not appear to observers to be research based. 
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Student Assessment 



Program guidelines originally required two forms of assessment: a norm-referenced test 
(ITBS) upon entry and in the spring and a periodic informal assessment. However, the 
ITBS requirement was removed from the guidelines early in the school year. Programs 
must keep records of student attendance, disciplinary referrals, books read, and reading 
achievement. 

Testing is the most prominent form of formal student evaluation at both public and 
private Reading Challenge sites in this study. Programs at school-based Reading 
Challenge sites generally receive BLT and ITBS scores from the school's administration 
of the tests. But programs housed in day care, church, or community center facilities do 
not have this benefit. Teachers in these programs generally administer the BLT or a 
substitute exam to students at the time of enrollment. Teachers at Site A report some 
problems with coordination of information between Reading Challenge and the school. 
They requested but received no information about students' literacy skills, learning 
disabilities, and behavior disorders. As a result, "the teacher did her own assessment." 

At Site B, the BLT "was given [at the home school] at the beginning of the school year 
and at the end, but the scores are not sent home or shared with the Reading Challenge 
program." The teacher adds, "BLT scores are recorded by the teacher for placement and 
progress [only]." Those programs that did not receive spring BLT scores from schools 
that administered the BLT on site in May to measure growth. 

A teacher at Site F does additional testing beyond the BLT. He looks at ITBS scores of 
the previous school year, administers a sight word recognition test, administers a context 
examination and also examines the Accelerated Reading Evaluation known as the STAR 
report. This one- to two-page report provides diagnostic information of general skills. 
Further, weekly writing samples are reviewed as an ongoing assessment at this site. 

All programs in the observation study use the BLT to determine or help determine 
individual student's reading needs. While some teachers complement the BLT with other 
testing procedures, few involve the student's classroom teacher in determining reading 
needs, although a few teachers do discuss children's reading progress with classroom 
teachers and make changes in the curriculum based on these mostly informal 
conversations. 

The assessment component of the Reading Challenge program is not consistent across 
sites. Sites report either using the BLT to assess reading proficiency or relying upon 
student’s home schools to obtain test information. Some Reading Challenge staff report 
having difficulty obtaining this type of information from the home school. Given the 
importance of student assessment in planning instruction and monitoring achievement, 
this component of the program may warrant further development. 
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Advisory Board 

Each Reading Challenge program must have an advisory board that is made up of a 
diverse group of representatives from school faculties, administrators, parents, and 
community leaders and is responsible for the following: 

• Becoming aware of all community resources. 

• Preparing a detailed outline of the program. 

• Approving beginning and ending dates of the program, as well as days and 
times of operation. 

• Developing a discipline policy and an attendance policy. 

• Meeting a minimum of two times during the year to discuss program 
improvement and evaluation. 

The advisory board tasks of networking and identifying community resources are 
reportedly invaluable to the success of connecting Reading Challenge to the community. 
Focus group participants reported that some board members are also involved in 
mentoring activities with the children. Other advisory boards are reportedly less effective 
and are not meeting the expectations according to program guidelines. Several sites 
lacked a discipline and/or attendance policy. Also, at least one advisory board is 
comprised only of Reading Challenge personnel, limiting their representation of parents 
and community members. 

Focus groups at eleven of the fifteen study sites discussed the effectiveness and impact of 
their advisory boards. Advisory boards were generally seen as a strength of the Reading 
Challenge program. They met regularly (once a month to twice a year) and helped to 
guide the program. The advisory board at Site CC assists with solutions to specific 
problems related to absences and discipline. At Sites CC and BB, the advisory board 
serves as a liaison to the community and helped to maximize resources. The group at 
Site H stated that their advisory board "helps us have a checks and balances system in 
place" to be sure the program is operated according to state guidelines. The group at 
Site AA described their active advisory board as "the trouble-shooting unit" in charge of 
ongoing evaluation and adjustment as the program operates, helping to further develop 
what has been put in place. The advisory board at Site AA meets at least once a month to 
review and assess Reading Challenge themes. The advisory board at Site B consists of 
the teachers and paraprofessionals employed by the Reading Challenge program. At their 
meetings, the lead teacher discusses individual students and their work, their strengths, 
and weaknesses that need to be improved. Members of the Advisory Board at Site E 
have observed classes, conducted a chess club, and provided an etiquette class for 
Reading Challenge students. 
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Advisory Boards at two sites (A, DD) seem to have little impact on the program. The 
focus group at Site A reported that an advisory board was established with educators and 
other professionals, but this group does not provide the support typically expected from 
an advisory board. There is no evidence that the board has met, and participants in the 
group interview were not aware of an advisory board. There was discrepant information 
regarding the advisory board at Site DD. The observer reported, “All building personnel 
interviewed felt that no functional advisory board ever existed, although the county level 
administrator we interviewed felt that the board was very effective. There is clearly a 
lack of communication between school and county personnel. In actuality, the building's 
instructional supervisor assumed the decision-making responsibilities of the advisory 
board. ” 

Sites G and C reported that, as of May 1999, they did not yet have an advisory board. At 
Site C, the YMCA program committee was serving as an interim advisory committee. 

The advisory board is considered a strength at many programs. Participants at Site F note 
that the diversity of the board members contributes to Reading Challenge. The board at 
that site includes "people from all walks of life in the community and socioeconomic 
levels," including a parent, a nutritionist, the assistant superintendent of schools, a 
stockbroker, a minister, the instructional lead teacher, the executive director of the Boys' 
Club, and a member of Family and Children Connection. 

Reading Challenge program guidelines (see Appendix B) specifically outline 
responsibilities of the advisory board which are essential to program operation. Sites that 
are aware of the boards’ presence and/or have active advisory boards view their role as 
critical to the program’s success. 

Communication 



According to program guidelines, parents must be informed of student progress and/or 
program activities on a monthly basis. Additionally, classroom teachers of Reading 
Challenge students must be involved in planning for student programs and informed of 
student progress on a monthly basis. 

School-based sites have a distinct advantage over private facilities for maintaining 
contact with classroom teachers. For instance, one report noted that “The teachers in the 
Reading Challenge program are part of the school faculty and have a comfortable and 
convenient mechanism for informal communication with other teachers, and in some 
cases, see the Reading Challenge students in their own classrooms. ” 

In most public sites, Reading Challenge teachers are also full-time teachers at the school 
and are acquainted with the classroom teachers of their Reading Challenge students. It is 
often possible for them to talk informally about students. School-administered BLT (see 
footnote, p. 6) scores are normally shared with school-based Reading Challenge sites, but 
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formal meetings between the two are rare, although some teachers report that a 
documented log of formal contacts is kept on file. Reading Challenge personnel at some 
sites send written communication to their students' schools on a regular basis. For 
example, at Site D, progress reports are sent out every two months, and notes to parents 
are sent out twice a month. Site BB provides weekly written reports to teachers and 
parents, and frequently gets verbal feedback from the teachers. At Site CC, both parents 
and teachers are advised of Reading Challenge activities through distribution of a 
monthly calendar. 

Communication between classroom teachers and Reading Challenge sites that are not 
school-based is more difficult. Some teachers receive no information from the schools 
regarding their Reading Challenge students' literacy skills or special needs, and they view 
this as a critical problem. However, other teachers report that they informally "check" 
with students’ homeroom teachers about homework and behavior problems. One private 
site holds weekly conferences with classroom reading teachers to discuss student 
progress. According to the observation report, “The teachers mentioned that efforts made 
to coordinate instructional practice [with the classroom teacher] had been met with 
resistance. It appeared as if there were some unresolved territorial issues between the 
schools and the program. Perhaps written, telephone, or on-site communications could 
be initiated by the Reading Challenge faculty in order to develop a more coordinated 
effort in the area of reading instruction.” 

The survey data, however, do not support the idea that public programs spend more time 
communicating with teachers. The data show that, on average, private sites spent an 
average of one more day each month communicating with fourth and fifth grade teachers 
than public sites. An equal amount of time is spent communicating with teachers across 
public and private sites for sixth and seventh grade students and on average, public sites 
spend one more day per month communicating with eighth grade teachers. The data are 
presented in Chart 7. 

Chart 7: Average Number of Days per Month Spent Communicating with Teachers 
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One explanation for this discrepancy may be the nature of the communication. It is 
possible that in public programs, communication is more informal and is not documented 
as it is in private sites. This will be more explicitly explored in the second year 
evaluation. 

Communication with parents is generally less frequent than communication with 
teachers. Some programs communicate with parents through informative flyers and 
newsletters, as well as through meetings. Others report sending progress reports to 
parents either monthly or bi-monthly. In cases where communication does take place, 
program staff feel that Reading Challenge has bolstered parental support. The survey 
data show that in both the public and private sites, Reading Challenge programs spend an 
equal amount of time communicating with parents each month. A group interview 
participant at Site C recommended that “administrative involvement from participating 
schools would be helpful to create a communication bridge” to best serve the students. 

Communication with the student’s home school teacher and parents is key to planning 
instructional activities. Some Reading Challenge programs, particularly private sites, 
report that the coordination with the home school that is necessary to address student’s 
needs is difficult to maintain. Public sites, where Reading Challenge teachers also teach 
during the day, seem to have the advantage of more frequent, but less formal 
communication. In these instances the opportunity to share information has been 
beneficial. Communication with parents is generally less frequent than communication 
with teachers. 

Staff Development 

Nearly all Reading Challenge teachers mentioned receiving helpful staff development 
training. Some teachers mentioned helpful staff development that was offered through 
their school rather than from Reading Challenge sources. 

Teachers at some sites, however, reported that they did not attend any Reading Challenge 
staff development activities. A teacher at Site A reports that she "was aware of staff 
development activities, but no one took advantage of them [and] no time was set aside for 
training." 

Some focus group discussants complained that they did not receive sufficient staff 
development. For instance, one participant said, "We need more planning before the 
program begins. . . . We need staff development as a site director, then the whole staff 
needs staff development and planning training." 

Transportation 

Issues related to student transportation were raised during the evaluation. Specifically, 
the issues of availability of transportation, safety and supervision on vans and buses, 
school liability, and bus driver responsibilities were a major concern. A comment that 
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reflected this concern was, “Transportation has been a real problem— we don’t have our 
own bus. We thought of car-pooling, but the insurance problem put us off that idea. We 
need more money to fix the transportation problem.” 

Group interview participants felt that “our lack of transportation adversely affects 
retention in our program.” The field observers at Site C noted that a resolution of 
transportation issues would “allow equal opportunity to all students” to participate in the 
Reading Challenge program. 
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Student Attendance and Achievement 
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This section of the report describes the attendance and reading achievement of students in 
the Reading Challenge program. It is important to re-emphasize the descriptive nature of 
the data, and to warn that summative conclusions should not be made based on this 
information. This first year report primarily addresses how the Reading Challenge 
program is being implemented, and which types of students are predominantly being 
served. Attendance and achievement data should be interpreted in this light. They, too, 
are most appropriately seen as describing student characteristics, and not as being a 
product of the Reading Challenge program. Program effects on achievement will be 
more thoroughly explored during the second and third years of the evaluation. 

Student Attendance 

Reading Challenge guidelines stipulate adherence to a strict attendance policy in order for 
students to remain enrolled in the program (see Appendix B). Field researchers reported 
major attendance problems at two out of 1 5 (in the observation sample) Reading 
Challenge sites. Site A was funded for 20 students, but one teacher reported an actual 
daily attendance of 5 to 7 children; sometimes as few as two students were present. 
Observers at the site report that teachers identify three reasons for “sporadic attendance.” 
These include students moving away, removal from the program for disciplinary 
problems, and student participation in other programs. 

Regardless of whether their attendance tended to fall below required minimums, retention 
of students was considered a problem at many Reading Challenge programs. For 
example, at Site CC, which had an original enrollment of 60 students, only 5 1 were 
enrolled at the close of the program year. However, only 21 of these students were part 
of the original 60. Advisory board members at Site BB expressed concern that they had 
only 20 of their original 30 enrollees. SiteAA reports 50 percent retention of original 
participants due to several factors. Speaking to this point, one board member said, “For 
some students who had not been used to daycare, it was too long a day; they missed 
home. Others had behavioral problems and were asked to leave. A few began a sports 
program that interfered with the Reading Challenge program.” 

Many programs reported having particular difficulty attracting and retaining middle 
school students. For these students, attrition was generally attributed to the availability of 
other extracurricular activities, especially sports. 

Chart 8 shows the average number of days students spent in the Reading Challenge 
program by program type. The data show that, on average, students in public programs 
spent more time in the program than students attending private Reading Challenge 
programs. This difference may be accounted for by date of program implementation, or 
may reflect differences in retention across the two types of Reading Challenges program. 
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Chart 8: Average Number of Days Students Participated in the Program by Type 
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The spring 2000 student focus groups will shed more light on this issue. However, it is 
clear that during the first year of Reading Challenge, neither public or private programs 
reported students participating for the full 130 day program. This compounds the 
difficulty of drawing conclusions about the effect of the program on students’ reading 
and general academic performance, and reinforces the need to view these data as 
descriptive of the students who participate in the program rather than the impact the 
program is having on individual students. 

Chart 9 shows the median length of time students spent in the program based on the 
following demographic characteristics: gender, race, and socioeconomic status (i.e., 
ffee/reduced lunch). These numbers suggest that white males from higher socio- 
economic families (i.e., students who do not receive free or reduced price lunch) 
generally spend the least amount of time in Reading Challenge. As noted earlier, 
students’ reasons for either persisting in or dropping out of Reading Challenge will be 
explored during the spring 2000 focus groups. 



Chart 9: Median Number of Days in Program by Demographics 
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The data on students’ attendance suggests that during the first year of implementation, 
Reading Challenge programs (both public and private) had difficulty retaining students 
and attracting older students. Students at public sites spent more time in the program 
than those attending private sites, however, neither group attended the program for the 
full 130 day duration. In terms of demographic characteristics, African American, lower 
socioeconomic students (i.e., free, reduced lunch) and females spent the most time in the 
program. 

Student Achievement 

In this section of the evaluation report the ITBS scores of fifth and eighth grade Reading 
Challenge students are used to describe their academic achievement. This group was 
selected because they were administered the ITBS as part of the statewide testing 
program in spring 1999. The ITBS achievement data is analyzed first by program type 
(public and private) and then by length of time in the program for each program type. 

Achievement by Program Type 

Fifth and eighth grade Reading Challenge student’s ITBS scores were examined to assess 
the extent to which program type (public vs. private) is associated with student academic 
achievement. Comparisons were then made by assessing impact on student achievement 
based on the length of time students actually participated in the Reading Challenge 
program. These analyses were performed to ascertain whether any different patterns of 
achievement exist among groups. 

Chart 10 shows the mean test scores by program type for the reading comprehension and 
reading total ITBS sub-tests for fifth grade students. Chart 1 1 shows the same 
information for eighth graders. Developmental standard scores are presented for the 
Reading ITBS subtests. Fifth grade students who chose to attend Reading Challenge at a 
private sector site typically had slightly higher ITBS reading scores than did their peers 
who attended public sector programs. In the eighth grade, this pattern was reversed, and 
far more pronounced. 

The difference between public and private sites, especially in the eighth grade, is 
particularly surprising in light of the fact that both public and private sites report 
spending comparable amounts of time on reading instruction. One plausible explanation 
for public school students having higher scores in the eighth grade may be that since most 
of the teachers at the public sites also teach at the students’ home school, they are better 
able to address directly the reading areas in which students need the most support. 
Another possible explanation may be that eighth grade students at public schools had 
higher achievement prior to attending Reading Challenge. This issue will be more fully 
addressed in the second year of the evaluation. 
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Chart 10: Mean Fifth Grade ITBS Scores by Program Type 
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Chart 1 1 : Mean Eighth Grade ITBS Scores by Program Type 
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Achievement by Length of Time in the Program 

Given the difference in date of implementation across Reading Challenge programs, it is 
also important to examine academic achievement relative to the amount of time students 
participated in the program. In order to examine ITBS scores relative to the length of 
time students actually spent in the Reading Challenge program, students were broken into 
three separate categories: low, medium, and high attendance. Each category represents 
the actual number of days students were in attendance. Students in the low category 
attended the program for one month or less (the least amount of time), students in the 
medium category attended between one and three months, and in the high category 
students attended the program for at least three, but not more than six months. Chart 12 
shows the percent of students in each of the time categories by program type. The data 
show that in both the public and private Reading Challenge programs, most students 
attended the Reading Challenge program between one and three months. 
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Chart 12: Length of Time in the Program By Type 
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ITBS scores were then examined relative to length of time spent in the program by grade 
level. These data are presented in Charts 13 through 16. 



Chart 13: Fifth Grade ITBS Scores by Program Length for Public Sites 
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Chart 14: Fifth Grade ITBS Scores by Program Length for Private Sites 



210 214 213 




Rdg Comp 



Rdg Tot 



□ low 

B medium 

□ high 



For both public and private Reading Challenge programs, the data show that the highest 
performing students spent between one and three months in the program. This coincides 
with the overall amount of time students in all grades attended the program. 
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Chart 15: Eighth Grade ITBS Scores by Program Length for Public Sites 
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Chart 16: Eighth Grade ITBS Scores by Program Length for Private Sites 
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For eighth grade, the data show that in private sites, the highest ITBS scores were earned 
by students who were enrolled in the program for three months or more. Conversely, in 
school-based Reading Challenge programs, eighth grade students who were enrolled for 
one month or less earned the highest ITBS scores. These differences may be a function 
of enrollment criteria. It may be that public sites (as indicated by the observation and 
focus group data) target low achieving students, therefore, high achieving students spend 
the least amount of time in the program. It is plausible that private programs have more 
diversity in terms of the type of student who is enrolled or that students’ retention in the 
program is not based on their academic ability. What this data means in terms of the 
program participants’ academic skills is unclear. These issues, too, will be explored more 
fully in the second year of the evaluation. 

Teacher’s Perception of Academic Achievement 

According to participants, the most dominant strength of Reading Challenge is the impact 
on student academic achievement. Teachers report that they see academic gains 
particularly in decoding and comprehension skills and which are often reflected in 
improved grades and understanding in social studies and science. Many focus group 
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participants agreed that, as one participant said, “The real strength of this program is the 
fact that reading is the core; if reading is strengthened, all areas are stronger. ” 

According to many participants, increased academic achievement leads to gains in 
motivation for learning. Teachers report that the confidence of Reading Challenge 
students has improved, and students are proud of their new knowledge. Participants 
believe increased self confidence has positive effects on classroom behavior and social 
skills, including respect for others. For example, at Site BB, participants say students' 
“self esteem is increasing and they are accomplishing more in their regular classroom 
setting.” Children are reading more books and have an improved attitude toward 
learning, and they are spending more time in the media center because of the program. 
Participants at Site FF echo this attitude improvement, claiming the Accelerated Reader 
Program motivates students to read more. 

Other sites use various means of assessment beyond the BLT to monitor student progress, 
including Accelerated Reader tests, oral reading, and various work samples. Many 
teachers report that improved student achievement has been observed in reading ability, 
reading fluency, and writing. Phonics, sight words, and identifying the main idea have all 
improved. Students’ attitude toward reading has also improved. In a group discussion at 
Site B, participants claim that "some report cards were better," adding that "they see 
evidence of improved self-esteem in the students." 

At Site AA, teachers report that reading skills have improved, but the students "have also 
blossomed in ways we did not expect." She says the children have "become more 
outgoing, raised hands in class, more self-confident, developed camaraderie, [and] some 
students who were shy are now less shy." These claims are supported, she says, by the 
technology coordinator and other classroom teachers at the school. A teacher at Site BB 
concurs, saying her students now enjoy reading. She commented, *7 think they feel more 
comfortable with their reading. Teachers at school have stopped me and told me they 
can tell a difference in our students, that they have come so far. That's when you know it 
works, when people besides us [Reading Challenge teachers] can see a difference." 

Some teachers stated that it is not easy to quantify the results of student progress because 
so much of it is in the affective domain. As one said, ", Progress is demonstrated through 
the willingness of the students to participate more in their regular classrooms; we can 
see the growth and improvement daily. ” 

One fifth grade student at Site GG, writing a report about Reading Challenge, eloquently 
defines progress in this way: "I like Reading Challenge because you can read when you 
want to.... I hope that when I go to CVMS (a middle school), lam allowed to go to 
Reading Challenge there, lam going to tell you some other good things about Reading 
Challenge.... Miss [teacher] has taught me to like to read because I hated to read 
before. Now I like to read. ” 
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Observers at Site F noted that, “Students appear to be highly motivated to participate in 
the RC program. According to all reports by Reading Challenge staff, children are eager 
to leave for their small group sessions each day and are increasingly enthusiastic about 
reading independently and within groups. Teacher-student rapport appeared very 
positive and upbeat. The learning environment was mutually supportive and engaging. ” 

The data presented on academic achievement is preliminary in nature. Although the test 
data do show some differences based on program type and length of stay in the program, 
with less than one year of implementation, it is not possible to determine whether these 
differences are related only to Reading Challenge. It is likely that Reading Challenge 
affects achievement indicators other than test scores that are not easily measured. 

Reading Challenge students may be benefiting from other educational programs that also 
enhance reading skills (i.e., Reading First, Title I, SIA, REP). Without isolating the 
impact of all of these services on academic achievement - a nearly impossible task in the 
real world of schools and schooling - it would not be possible to attribute changes in 
student performance to Reading Challenge. Given that the impact of Reading Challenge 
on students’ test scores are difficult to measure, the program observations and 
testamonials of Reading Challenge and home school teachers become even more 
important. Teachers’ belief that Reading Challenge students are improving their reading 
skills and their attitude toward reading is almost unanimious. 
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Part IV: 

Summary and Conclusions 

The first year evaluation of the Reading Challenge program is mainly descriptive in 
nature. This is appropriate for the evaluation of a new, statewide program. Data from 
observations of 15 purposively chosen sites as well as focus group and survey data were 
used to examine how the program is being implemented and to provide preliminary 
information on program effects on achievement. 

In terms of implementation practices, the data showed that, for the most part, Reading 
Challenge is being implemented as planned. Reading is the cornerstone of the program 
and the data did show that both public and private sites spent more than the required 
amount of time on reading instruction. However, there were some slight differences 
between public and private sites. For instance, there was variation in the criteria used to 
select Reading Challenge participants. Public sites tended to select students based upon 
low achievement in reading/language arts, while private sites tended to either open the 
program up to anyone on a first-come-first-served basis or targeted a select group (i.e., 
Hispanic, transient students, or children of the church implementing the program). There 
was also some slight difference in public and private programs in attendance and reading 
achievement, but these were either not significant or may be understood only through a 
more long-term examination. Both public and private Reading Challenge programs 
indicated that it was difficult to keep students in the program for the lull 130 days. There 
is also evidence that daily attendance is a problem at some sites. It will be difficult to 
assess student achievement if many students do not participate for the frill program 
period. Finally, the preliminary data on achievement is mixed. Overall, students 
attending private programs had slightly higher ITBS scores in fifth grade, however, 
students attending public programs showed higher ITBS scores in the eighth grade. 

When scores were assessed relative to the length of time in the program, the results were 
mixed between fifth and eighth grades and between public and private facilities. Without 
pre-implementation data, which show how students performed before entering the 
program, it is difficult to understand what these results might mean. In addition, the 
problems assessing program effects of only a small portion of a student’s total 
educational experience makes interpretation of the first year achievement data difficult. 
Finally, teachers' perceptions of students' academic growth were positive. Teachers felt 
that students' academic performance is improving as a result of the program. 

Data collected by way of the focus groups, surveys, and classroom observations resulted 
in a number of specific recommendations. Some of those with most agreement among 
Reading Challenge staff, many of whom are also the home school teachers of these 
students, include: 

1 . Reduce the number of specific program goals to the most important ones 
given the short time for instruction and the circumstances of an after-school 
program. 
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2. In reducing the goals for all programs statewide, encourage individual 
programs to tailor their local goals and program implementation to identified 
local needs. 

3. Provide more state guidance/training in the following areas: 

• effective ways to meet objectives and implement all components of the 
guidelines. 

• alternative staffing patterns to reduce teacher burnout. 

• strategies to attract and use volunteers. 

• lesson planning with an emphasis on how to design activities that will 
work well under the special conditions of a voluntary, after school 
program. 

4. Provisions (in guidelines and budget) for planning and training prior to 
beginning each year’s program. 

5. Guidelines should place more emphasis on integration of children’s literature 
and explicit guidelines for reading instruction in the program. 

6. Examples of recommendations for state guidance on instructional practices 
included: 

• implementing community activities that require reading. 

• providing instruction in higher level comprehension. 

• emphasizing writing process instruction. 

• developing thematic study approaches to instruction. 

• teaching reading comprehension strategies. 

• integrating science and math. 

7. Encourage the inclusion of community activities that require reading in the 
content area as an integral component of the program for all ages. 

8. State conference(s) to allow all Reading Challenge sites to share methods and 
techniques. 

9. Budgetary provision for sites that have transportation problems. 

10. Provide a standardized, paper and pencil reading test that can be used with all 
students on a pre-post basis for assessment of individual reading achievement. 

The evaluation of the Reading Challenge program will continue during the 1999-2000 
and 2000-2001 school years. New sites will be added and existing Reading Challenge 
programs will continue to be followed. Data will be collected from administrators, 
teachers, and students to provide a broader perspective and more detailed information on 
the program. 
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Appendix A 



Methodoloev 

Protocol for Pre-Observation Teacher Interview 

Source: Research, Evaluation, and Testing Division, GDOE 

Guide for Classroom Observations 
Source: University of Georgia, ORG 

Observation Checklist 

Source: University of Georgia, ORG 

Protocol for Group Discussion/Interview 
Source: University of Georgia, ORG 

Readiness Challenge Survey 

Source: Research, Evaluation, and Testing Division, GDOE 
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Appendix B 



Reading Challenge Program Guidelines 
Source: Curriculum and Reading Division, GDOE 

Note: If you require hard copies of any of theses appendices, please contact Laura Green 
at the Georgia Department of Education, lagreen®doe . k!2 . ga . us or 404-656- 
2668. = 
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